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SSAT Project Objective (from NASA PRG) 



Objectives: The System-Wide Safety and Assurance 
Technologies (SSAT) project will identify risks and 
provide knowledge required to safely manage 
increasing complexity in the design and operation of 
vehicles and the air transportation systems, including 
advanced approaches to enable improved and cost- 
effective verification and validation of flight-critical 
systems. 

The Project will address the following challenges: 

* [Develop] verification and validation tools for 

manufacturers and certifiers to use to assure flight 
critical systems are safe in a rigorous and cost- and 
time-effective manner. 

• [Understand and Predict] system-wide safety 

concerns of the airspace system and the vehicles by 
developing technologies that can utilize vehicle and 
system data to accurately identify precursors to 
potential incidents or accidents. 

* [Understand] the key parameters of human 
performance which provide the human contribution 
to safety in aviation. 

• [Predict] the [remaining useful] life of complex 
systems by reasoning under uncertainty about root 
causes (diagnosis) and predict faults and remaining 
useful life (prognosis) across multiple systems. 


NASA 

Aeronautics Research Mission Directorate 

FY 2012 Planning. Programming, 
Budgeting and Execution Process 

Program and Resources Guidance 
May 6, 2010 


Project Reorganization 


SSAT Project 

System-wide safety 


IVHM Project 

(2007-2010) 


Aviation Safety Program 

Integrated Vehicle Health Management (IVHM) Project 





Validated, proactive solutions for ensuring safety in flight and operations 



VSST Project 

Vehicle systems safety 




Aviation Safety Program 

Integrated Vehicle Health Management (IVHM) Project 

Dr. Ashok Srivastava, Principal Investigator 
Dr. Robert Mah, Project Scientist 
Robert Kerczewski, Acting Project Manager 






IVHM Project Goals 


“Develop technologies to reduce accidents and incidents by 
developing vehicle health management systems to determine the 
state of degradation for aircraft subsystems; developing and 
demonstrating tools and techniques to mitigate in-flight damage, 

degradation, and failures” 



IVHM Research Framework 



Level 4 - Aircraft Level 


Goal - Validated multidisciplinary integrated vehicle health management tools 
and techniques to enable automated detection, diagnosis, prognosis and mitigation of 


IVHM 4.1 Vehicle-Level 
Reasoning and Ground/ 
Right Test Evaluati ons 


IVHM 4.2 Systems 

adverse events during flight. 

IVHM 4.3 

IVHM 4.4 Research 

Analysis 


Dash link 

Test and Integration 



IVHM 3.1 


IVHM 3.2 


IVHM 3.3 


IVHM 3.4 


IVHM 3.5 

Detection 


Diagnosis 


Prognosis 


Mitigation 


Integrity Assurance 



Level 2 - 
Subsystems 


IVHM 2.1 Aircraft 


IVHM 2.2 


IVHM 2.3 


IVHM 2.4 

Systems HM 


Airframe HM 


Propulsion HM 


Software HM 



Level 1 - 
Foundational 


IVHM 1.1 Advanced 
Sensors 
and Materials 


IVHM 1.3 Advanced 
Analytics and 
Complex Systems 


IVHM 1.4 Verification 
and Validation 


Technology Level/Fiscal Year 08 09 


Multidisciplinary IVHM Technologies, Tools, ^ Document and Disseminate One Ground-Based Test A One Flight Test A 

and Techniques Technologies for Research 


Systems Analysis for Health Manage ment 


Discovery in Aeronautics Systems Health Internal A A External DASHIink Implement 
/r\AcultnL\ DASHIink r £ rs~ rs f 



Implement. ^ ^ ^ — 3 ~ 

Research Test and Integration 

A Review So A in Integration a ^ <1AA1A ,~ 

Convene IAAWG AA Develop Integration Strategy Document FmdlngS ° f ' AAWG 

Detection 

Baseline Assess of Capabilities O Assess of Valid. Demos (3 of 5 Adverse Event Types) w/ Improve. O 

^ __ — Fleet-Level Anomaly Detection 

Diagnosis 

Baseline Assess, of Capabilities ♦ Assess of Valid Demos.^3 of 5 Adverse Event Types) w / Improve. A Auto. Diag. Capblty 

+ + + + Demo of Disambig. Sub-sys Faults 

Prognosis 

Algorithm Quality Method A | ^^Forecast. Tech, for 

Baseline Assess, of Capabilities <x> Fidelity Guideline Assess of Prog. Reasoning Ability O Anomaly Predict. 

Mitigation 

Estab. Min. Perform. Criteria of ♦ ♦ Assess. Demo, of Mitigation for 

Candi date Miti gati on Strategi es at Least 2 Adverse Events 

Integrity Assurance 

Baseline Assess, of V&V <> Demo. 80% of Required Testbeds and 

Capabilities Meet 95% of Requirements for Each ^ 

Aircraft Systems HM 

Validate Method, and Tools 

Mitigation of Flight Computer and + + + for Failures Prognosis 

Actuator Failures and Damage Lightning Tools and Techniques ♦ ♦ Validate Method, and Tools 

Airframe HM 

Validate Method, and Tools for Diaanosis _ , ., N . ♦ Validate Method, and Tools 

y Demo. Multiple Sensor Tech. O C> OfnrPmnnncic 

IRAC/IVHM Ground-Based Demo a ^ + ▲ ^ A ~ ror prognosis 

Flight Data Acquisition Z ♦ ♦ T_ -♦ Demo. Self-Healing for In-Situ 

Propulsion Systems HM 

Demo High-Temp Wireless Sensing Sys.O Demo. Multi .Sensor 0 <> Demo High-Temp Wireless Sensing Sys. 

♦ ♦ Validate Methods and Tools Technologies ^ ♦ Demo. Multi. Hioh-Temc. Sensors (Gas Path) 

Software HM 

Initiate SoA Survey O Consistent Evidence O O 0 Eval. of Integrated O 

Accum. Framework S/W Malfu net Classification O Adapt. Reconfig. 

Advanced Sensors and Materials 

User Requirement Document <§> ^ High-Temp Power ^Optical Propulsion HM Demo. 

Physics-Based Models Demo. ♦ $0 HM°Nano 4 Se^ KST Z ° ° ^ SenSI " 9 D<5m °' 

Modeling 

User Requi rement Document^ Alqorithm Develop . ♦♦♦ ♦ 

Testbed Failure Metrics Develop^'O' O"<0 O Valldate Models for Electronics 

♦ O < ^ > ^“Develop Bayesian Method, and Hybrid Reason. Tech. 




Implmnt. & Bench. Improved Algorithms for Fault Diag^ O Offline Mode Auto. Anomaly Detect. Demo. 

Advanced Analytics and Complex Systems Establish User Requirements <§> <> ~ ^^AOOO <j> ^ Implmnt. & Bench. Decision- 

Real World Data Acq.O O+O Implmnt. & Bench . Reconfig. Algorithms'^ ♦ O Theor. Algorithms 


Verification and Validation 

Compositional O O 

Verification Demo. * ^ - -<^> — -<*>— Formal Verification and 

Automated Testina Demo. 

m Level 4 A 

Recurring A 

DetectionO Diagnosis^ PrognosisO Mitigation^ Integrity AssuranceO 






IVHM NRA Partners 



o 



f m li N t V 1 R S 1 T V of 

Ur FLORIDA 


H 

u 



Arizona Stati 
University 



UNIVERSITY 




STANFORD 

UNI Vt RSITV 



University 
of Minnesota 


• Our Portfolio 

- On-board system failures and faults - 3 active 

- Detection -- 6 active and 1 completed in FY10 

- Diagnosis -- 7 active and 1 completed in FY10 

- Prognosis -- 6 active and 1 completed in FY10 

- Mitigation -- 2 active 

- Integrity Assurance -- 5 active 

- Ongoing monitoring of operational data -- 3 active 

• Tracking Progress 

- All reviews are conducted annually at the Project Level: PI, PM, PS, API, COTR/TM + other interested 
parties 

- Reviews are conducted via WebEx 

- Review comments are formally collected and forwarded to awardee via COTR/TM 

- Many face-to-face interactions occur annually at both NASA and awardee sites 

- All NRA documentation is stored on NX so that the entire project team has access 

NRA Value to IVHM 

- Overall the performance of the NRA awards were judged VERY GOOD. 

- Each award is mapped to one or more approved IVHM Technical Plan milestones. 


galois 





, si 

Impact/ 


systems « 


Cotter Research Corporation 


Honeywell OAI 

W CbtiAmiwWi 




Uftretuo w \ 


RitJgctop * 

Group im 


LUjNAV 
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SSAT Project Goals 


• "Understanding and predicting system-wide safety concerns of the airspace system ...and 

the vehicles as envisioned by NextGen, including the emergent effects of increased use 
of automation to enhance system efficiency and performance beyond current, human 
based systems, through health monitoring of system-wide functions that are 
integrated across distributed ground, air, and space systems.... 

• Develop fundamentally new data mining algorithms to support automated data analysis 

tools to integrate ... from a diverse array of data resources" 

• "Research to improve confidence and timeliness of certification... " 

• "Develop improved system engineering processes and tools for determining optimum 

roles of humans and automation in complex systems..." 


• Applied Research on Complex Systems Validation and Verification 

• Applied Research on Vulnerability Discovery 

• Applied Research of Human Performance Models 

• Applied Research on System Health Management 
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SSAT Research Framework 


Level 2 - Project Level 


Goal - Develop validated multidisciplinary tools and techniques to ensure system safety in 
NextGen to enable proactive management of safety risk through predictive methods. 


SSAT 2.1 Technical 

SSAT 2.2 Systems 

SSAT 2.3 Partnerships 

SSAT 2.4 Research 

Challenges 

Analysis (SA) 

and Outreach 

Test & Integration (RTI 


Level 3 - Subproject 


SSAT 3.1 


SSAT 3.2 


SSAT 3.3 
Human Systems 
Solutions (HSS) 


SSAT 3.4 

Verification & 


Data Mining and 



Prognostics and 

Validation of Flight Critical 


Knowledge 



Decision Making 

Systems (VVFCS) 


Discovery (DMKD) 



(PDM) 


Level 4 - Subproject Elements 


• SSAT 4.1.1: Argument- 


• SSAT 4.2.1: System- 

Based Safety 


Level Reasoning 

Assurance 


• SSAT 4.2.2: Anomaly 

• SSAT 4.1.2: Authority 


Detection from 

and Autonomy 


Massive Data Streams 

• SSAT 4.1.3: Distributed 


• SSAT 4.2.3: Discovery 

Systems 


of Causal Factors 

• SSAT 4.1.4: Software 


• SSAT 4.2.4: Prediction 

Intensive Systems 


of Adverse Events 


• SSAT 4.3.1: Human 


• SSAT 4.4.1: Decision 

Automation Tools 


Making under 

• SSAT 4.3.2: 


Uncertainty 

Operational 


• SSAT 4.4.2: 

Complexity Metrics 


Diagnostics 

and Methods 


• SSAT 4.4.3: 

• SSAT 4.3.3: Human 


Prognostics 

Performance 


• SSAT 4.4.4: Software 

Mechanisms 


Health Management 


"Validated, proactive solutions for ensuring safety inflight and operations" 
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SSAT Project Technical Challenges 



1. Assurance of Flight Critical Systems (FY25) 

Development of safe, rapid, and cost 
effective NextGen Systems using a 
unified safety assurance process for 
ground based and airborne systems. 


2. Discovery of Safety Incidents (FY19) 

Automated discovery of previously 
unknown precursors to aviation 
safety incidents in massive (>10 TB) 
heterogeneous data sets. 



3. Automation Design Tools (FY20) 

Increase safety of human - 
automation interaction by 
incorporating human performance 
considerations throughout the design 
lifecycle in NextGen technologies. 


4. Prognostic Algorithm Design for 
Safety Assurance (FY25): 

Development of verifiable 
prognostic algorithms to help 

remove obstacles to certification. 




Technical Challenge 1 
Assurance of Flight Critical Systems 



Safe and Rapid Deployment of NextGen 

Fill a critical gap in the life-cycle 
development of complex systems for o 

NextGen by developing time- and cost- jj 

effective techniques for verification and x 

ifi 

validation of complex civil aviation systems 0 

that will unify processes for ground based Z 

and airborne systems (FY25). 0 

o 

Benefits: > 

• Rapid but safe incorporation of | 

technological advances in avionics, $ 

software, automation, and aircraft and ^ 

airspace concepts of operation. 

• Availability of safety assurance methods 
for confident and reliable certification, 
enabling manufacturers and users to 
exploit latest technological advances and 
operational concepts. 


209 . 00 1 

175.00 ■ 

150.00 - 

125.00 - 

100.00 - 

75.00 - 

50.00 - 

25.00 - 
0.00 - 

Requirements Design Code Development Acceptance Operation 

Test Test 

Phase in which error was detected and corrected 

Boeing 787 software cost ~$4.5B 



Technical Challenge 2 
Discovery of Safety Incidents 



Automated discovery of previously unknown 
precursors to aviation safety incidents (FY19). 

A first-of-a-kind demonstration of the 

automated discovery of precursors to aviation 
safety incidents through analysis of massive 
heterogeneous data sets. 

Benefits: 

• Understanding the impact of degradations in 

human performance on aircraft performance. 

• Identifying fleet-wide anomalies due to 

mechanical and other related issues that can 
impact safety, maintenance schedules, and 
operating cost. 

• Development of advanced methods to 

predict adverse events due to introduction of 
new technologies in NextGen. 



Sample Text Report 

JUST PRIOR TO TOUCHDOWN, 
LAX TWR TOLD US TO GO 
AROUND BECAUSE OF THE 
ACFTIN FRONT OF US. ... 
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Example Applications on ISS 




IMS 



t 

| |l |l *1 t 

*1 'i ; i ’i • 

SyntaModding 



Automatically learns how the system typically behaves 
and tells you if it is behaving differently now 


• Control Moment Gyros 

• RGA 

• ETCS 

• ARJ 

• Beta Gimbal Unit 

• CDRA 
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ISS Early External Thermal Control System 


ISS Early External Thermal Control System (EETCS) 



* EETCS used to dissipate neat on-board ISS 

* Heat transferee to liquid arrmoria cooling oops 
» Ammonia circulated to external radiators to cool 

• In early January 2007 EETCS developed an 
ammonia gas bubble 

• Bubble noted by ISS cont'ollers"9 hours oefore 
it 'burst' and dissipated back into liqu d 


Results: ISS Early External Thermal Control System 


ffKS Cm* (Ii»hhi.i a 1)1 - rmi mifu ta. if no;. 


14 

1J 



a ■ 




jr* u T 

point change 


bvibble 
via nor#r*l 

tclomctfv 

4 % * 7 • 

fr*r - 


Initial IMS indications 
~ 6 days prior to 
detection via standard 
techniques 




fc c-pins Co grOkV 


Arrvr»on i* 
b-gbfcl* — 

tniiS '5 


IMS trained on 185 days of data collected June • December 2006 
23 parameters analyzed (pressures, temperatures, quantities, pump speeds) 
2 -score normalization, no external eomputations/derived parameters 
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Example Application on STS 


STS-107 Columbia Ascent IMS Analysis 



O fcrmor 


♦ Trained on telemetered date 
from 10 previous 
Columbia flights 


• Data vectors formed from 4 
temperature sensors 
Inside the wing 


Mi(50i/ttid WhM r. m .|. 


♦ Data covered firsts minutes 
of each fl ght Munch to 
Mam Engine Cot Off’ 


Normalization: 

• Data expressed as value 
relative to a reference sensor 
(MLG Outboard Wheel Temp] 
to account for wide ambient 
temperature variations In 
training data 


UpfP»f W i fig Wl«v Trrr fi 


I tnfcdti 


i AUiMUi kn^ 


STS-107 Launch IMS Analysis 

« - — 
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Example Application on STS 


Space Shuttle Wing Leading Edge Impact 
Detection System (WLEIDS) 



132 1-D accelerometers 
mounted on the w;ng spar 
behind RtC panels 

20 KHz sensor data 
collected dLring ascent 

Once on orbit, sensor data 
summary files transmitted 
to Mssion Control for 
analysis 


Orca/iMS vectors constructed from S sensor values, Including a target sensor and 
surrounding sensors that might pick up radiating impact energy 



* WinpU-JilingEiigt* 

Pjraii Jir-d tancart 


f 



OET |s) 


Points of Interest 
Detected by Orca/IMS 
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Technical Challenge 3 
Automation Design Tools 


Advancing Safety by Understanding 
Human Performance 

Develop analysis tools that incorporate 
known limitations of human performance 
and enable design of robust human- 
automation systems to increase safety 
and reduce validation costs in NextGen 
(FY 20). 

Benefits: 

•Methods and tools appropriate for 
designers, trainers, and operators. 
•Enable the prediction of human 
performance to identify, evaluate, and 
resolve safety issues due to Human - 
Automation interaction. 



Graphic Display *) 




Technical Challenge 4 

Prognostic Algorithms for Safety Assurance 



Prognostic Algorithm Design for Safety 
Assurance 


Development of a new class of 

verifiable prognostic algorithms to 
help remove obstacles to the 
certification of prognostic algorithms 
(FY25). 

Benefits: . 

• New class of verifiable systems 
health management algorithms and 
methods. 

• Lowered barrier to deployment of 
systems health management 
algorithms. 


Edge 540T Flight Test bed 




BHM hardware & 
Real time CPU 


Particle Filter estimate 


Real time particle filter 
for battery RUL prognosis. 


Expected times of 
reaching threshold 
(darker lines denote 
later predictions) 



i 



RUL prognosis algorithm 
Implemented in Simulink 


I 


1 5* 
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SSAT Technical Challenges Cover a Broad Range of 
Safety and Assurance Technologies 


A Relevant probable 
causes: 

(1) Electrical bus failure 
resulted in loss of 
cockpit display and 
other functions 


Addressing Issues to 
Enable Certification 


Integrity 

Assurance 


Addressing Issues 
to Enable Discovery 
of Safety Issues 


Focus on Humans and Airspace Related Systems 



▲ 

Assurance of Flight 
Critical Systems 

A 

Prognostic Algorithms 

A 

Automation Design Tools 


A 

Discovery of Safety Issues 

Single Aircraft 

Multiple Aircraft, Machines, and 


CO 

cy 
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_o 

O 
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_c 

u 

,(D 


O 

>* 

-M 

CD 

H— 

03 

IS) 

CUO 
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s— 
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CO 

3 

u 

O 



/ \ Relevant probable 
causes linked to V&V: 

(1) ADIRU provided 
erroneous data 

(2) Flight control 
computers did not 
filter data. 


/\ Relevant probable 
causes linked to HAI: 

(1) Human- 
performance and 
workload 

(2) Human-automation 
interaction. 


/\ Relevant probable 
causes: 

(1) Impaired 
performance from 
fatigue and situational 
stress 

(2) Maximum cross- 
wind component 
exceeded. 

(3) Inappropriate use 
of reverse thrusters 


Humans 


Safety Coverage 
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SUB-PROJECT LEVEL PROJECT LEVEL 


SSAT Project Organizational Structure 


Technical 

Challenges 


Project Manager, Ashok N. Srivastava, Ph.D. 
Deputy Project Manager, Jessica Nowinski, Ph.D. 
Project Scientist, Robert Mah, Ph.D. 


I 


Deputy Manager 


Deputy Manager 


Deputy Manager 

(DPMF) for ARC 


(DPMF) for DFRC 


(DPMF) for GRC 

N/A 


Leslie Molzhan 


Amy Jankovsky 


Systems Analysis TL 


Partnerships TL 


VVFCS TL and IM 


DMKDTL 


Human Systems TL 


PDMTL 



Systems Analysis 


Gaye Graves 


Guillaume Brat (IM) 


Nikunj Oza 


Mike Feary 


Kai Goebel 


Business Team 



▼ 

Systems Analysis 


VVFCS Testbed 

VVFCS 



Data Mining 


Prognostics 


Business Team 

Business Team 



Deputy Manager 
(DPMF) for LaRC 
Debbie Martinez 

Systems Analysis 


Eric Cooper 
Paul Miner 


Kara Latorella 


Prognostics 


Business Team 
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SSAT Partnership Strategy 


SSAT develops partners based on a strategic need (as assessed by the Project Management Team) in 
the following areas: 

• Access to data not readily available to NASA that is directly related to a Tech Challenge 

• Experimental platforms and unique expertise directly related to a Tech Challenge 

• Unique test, integration, and infusion opportunities 

We are frequently approached for potential partnerships from domestic and international 
government agencies, academic institutions, air carriers, and major industry players. 


as 0 

easyJet 

Validation of data mining algorithms for discovering 
precursors to aviation safety incidents. 


Research Test and Integration Collaborations 

SSAT CoodrTch 

• Partial list of partners supporting collaborative research 

MAKEL vs ST afr^ 

• Prognostic algorithms for EMA; integrated research on 

^ Pratt & Whitney ^ y •* ^j| S 

Engine Fault Detection and Diagnosis 

SSAT RTI Partnerships and Outreach 

• V&V and Software Health Management 


• Pilot fatigue (SOFIA, Air Force) 


• Support research in Airspace Concepts 
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Overview- SSAT Partnerships (II) 



Assessment of current Systems Health Management capabilities and 
emerging technologies for V&V, Data Mining, Human Automation 

and Interaction Tools, and Prognostics/Decision Making; 

development of an analytical framework for evaluation and 
benchmarking of these technologies; and collaboration in health 
management data and algorithms. 

VSST / AEST 

• System architecture to enable resilient flight deck automation 
technologies based on the output of the Vehicle Level Reasoning 
System. 

• Vehicle level detection and diagnosis of sensor and actuator faults; 
application of virtual sensor technology; system architecture to 
enable resilient adaptive control based on the output of the Vehicle 
Level Reasoning System. 

AFRL» 

•Vehicle-level architecture and reasoner 

•Ground to flight architectures and testbeds M Bk 

• IVHM-enabled CBM 

• Data Mining 
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Overview - SSAT Partnerships (III) 


easyJet ^_S NER ^, 

THE FtENCH AEtOSEACE LAI 

Validation of methods to discover precursors to aviation 
safety incidents and the impact of pilot fatigue. 

£r'j The Joint Planning and Development Office 1 

Making NextGen a Reality 1 

Or ** 

Cooperative research and technology development 
(R&TD) activities in the areas of V&V, data mining, and 
human automation and interaction tool technologies and 

Administration 

systems. 

s 

Prognostics of composites. (SAA) 

STANFORD 

UNIVI RMT% 


Airspace Systems Program 

Co-funding CMU NRA for demonstrating compositional 
verification on separation assurance software 

Networking and Information 
Technology Research and 
Development Source (NITRD) 

Participation/representation for three NITRD Program 
Coordination Areas: High Confidence Software and 
Systems; Software Design & Productivity Human 
Computer Interaction & Information Management 

Joint Safety Analysis Team (JSAT) 

Year long collaboration and membership regarding the use 
of data mining to discover precursors to safety incidents 
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SSAT Research Partners 



Assurance of Flight Critical 
Systems ( including 
Software Health 
Management) 



Kestrel 

Technology 



& 


( iiriM^gic Mellon 


Honeywell 


I'lir 


Massachusetts 
Institute of 
Technology 





galois 



ISR 


Discovery of Safety Issues 


Honeywell 




Automation Design Tools 


MICHIGAN 


The fe 
University 
of Iowa 


Prognostic Algorithm 
Design for Safety 
Assurance 



# ^ 

Impact/ 


• 5 * .tffk 


STANFORD 

uni vi rsit v 





48? i 


AUBURN UNIVERSITY 
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Progress Metrics for SSAT Research 
A Model-Based Approach 



• SSAT used a model-based approach to assess the impact of our research and progress 

toward meeting our TC. Uncertainty of progress metric increases with time. 

• The assumptions have been validated with the Technical Leads and DPMFs. 

• These metrics give only one assessment of the progress towards solution of the challenge. 
There are other ways to demonstrate the progress and impact of our research. 

• Models incorporate an assessment of probability of technical infusion, thus helping to 
address progress towards completion of TC. 

• About the Models 

- Model parameters can be changed based on new information and can be used to perform 'what-if 
analysis', such as, 'what if our research produces a 20% improvement in accuracy instead of a 10% 
improvement?'. 

- The models include factors that are 'hard-benefits' such as improvements in accuracy, speed, etc., 
and 'soft-benefits' such as 'improvement in query technologies'. 

- The models include a parameter that assess the likelihood of technology transition into a real- 
world implementation (not just transition from NASA to industry). 

- The models are tied to overarching safety goals with specific Aviation Safety incidents and 
accidents cited using an approach similar to that used in the IT industry. 

• SSAT will update these models routinely to maintain relevance to Tech Challenges and 
changing research results and needs. 


All models are wrong, but some are useful- G. E. P. Box 
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Progress to Completion of Technical Challenge: 
Assurance of Flight Critical Systems 


Progress to Completion of Technical Challenge 1 
Assurance of Flight Critical Systems 



1 Baseline 

2 Static code techniques for certification 

3 Analytical framework for mitigation strategies 

4 Use of formal methods as evidence for safety cases 

5 Compositional reasoning as verification techniques 

6 Formal models for analyzing human/automation roles and responsibilities 

7 Prototype of integrated tool for resilience engineering for integrated distributed 
systems 

8 Advance safety assurance to enable deployment of NextGen flight critical systems 


Measuring Progress 
Assurance of Flight Critical Systems 



FY 11 

FY 12 

FY 13 

FY 14 

FY 15 

FY 16 - 30 


FY12Q4 Analytical FY14Q3 Compositional FY14Q3 Formal models for 

framework for reasoning as verification analyzing human/ 

mitigation strategies techniques automation roles and 

responsibilities 

FY12Q2 Static code FY13Q1 Use of formal FY15Q3 Prototype of integrated tool for Resilience 

techniques for methods as evidence for Engineering Integrated, Distributed Systems 

certification safety cases 

FY15Q4 Advance safety assurance to enable deployment 
of NextGen Flight Critical Systems 

What are the intermediate and final exams to check for success? 

• Demonstration of a 0% false positive rate by combining static analysis and model checking 

• Development of validated communication topologies 

• Unified approach to autonomy and authority 
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Progress to Completion of Technical Challenge: 
Discovery of Safety Issues 


Progress to Completion of Technical Challenge 

Discovery of Safety Issues 




1 Baseline 

2 Scalable algorithm for anomaly detection on heterogeneous data 

3 Scalable algorithm for prediction of prescribed adverse events in discrete 
and continuous data 

4 Vehicle Level Reasoning 

5 Identification of precursors in flight and text data 

6 Automated discovery of precursors to safety incidents 
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Measuring Progress 
Discovery of Safety Issues 



FY 11 

FY 12 

FY 13 

FY 14 

FY 15 

FY 16 - 30 


FY11Q4 Scalable algorithm FY12Q4 Scalable algorithm for 
for anomaly detection on prediction of prescribed 
heterogeneous data adverse events in discrete 

and continuous data 


♦ — 4 ♦ ♦ — ♦ 

FY13Q2 Vehicle Level Reasoning FY14Q4 Identification of 

precursors in flight and text data 


FY15Q4 Automated 
discovery of precursors to 
safety incidents 


What are the intermediate and final exams to check for success? 

• Development of methods to analyze 10 TB of heterogeneous data 

• Development of methods to identify crew performance degradation 

• Development of predictive methods for heterogeneous data sets. 
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Progress to Completion of Technical Challenge: 


Progress to Completion of Technical Challenge 

Automation Design Tools 



2 Methods for determining functional state in operations 

Develop technologies to provide early detection and mitigation of flight crew 

3 performance issues, using unobtrusive behavior monitoring. 

4 Tools for evaluation of human - automation procedural complexity 

5 Predictive Human Performance Design Tools 

Develop toolbox and guidelines for incorporating multimodal information 

6 management strategy 

7 Identification of novel Human-Automation Interaction Failures 

8 Human Automation Design Tools 

Aviation Safety Program Annual Review November 16-17, 2011 | SSAT Project 


Measuring Progress 
Automation Design Tools 



FY 11 

FY 12 

FY 13 

FY 14 

FY 15 

FY 16 - 30 


FY12Q4 Methods for 
determining human functional 
state in operations 


FY15Q4 Identification of novel Human 
-Automation Interaction failures. 
Human Automation Design Tools 


♦ ♦ ♦ 

FY14Q4 Predictive Human 
Performance Design Tools 


What are the intermediate and final exams to check for success? 

• Proof-of-concept tools demonstrating the ability to support the design validation and verification 
process; Framework reviewed by subject matter experts. 

• Proof-of-concept Matlab based visualization tool suite for monotonic analog signals arising from 
sensor and performance based aircraft operations or faults. 
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Progress to Completion of Technical Challenge 
Prognostic Algorithms for Safety Assurance 



oi 



1 Baseline 


2 Performance baseline for prognostic algorithms 


3 Safety Assurance performance metrics for prognostic algorithms 

4 Demonstrate mission extension 

5 Integrated Decision Making 


6 Demonstrate avoidance of mission abort 

Demonstrate verifiable prognostics on flight vehicle 


7 
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Measuring Progress 

Prognostics Algorithms for Safety Assurance 


FY 11 

FY 12 

FY 13 

FY 14 

FY 15 

FY 16 - 30 


FY13Q1 Safety assurance FY15Q4 Demonstrate verifiable 
performance metrics for prognostics on flight vehicle 
prognostic algorithms ssat.i.i.pdm.3.05 

Prognostics Demonstration 

FY11Q4 Performance baseline 
for prognostic algorithms 



What are the intermediate and final exams to check for success? 

• Demonstrate the prognostics algorithm meets the verifiability metric previously identified, 
and demonstrate using a flight vehicle that the previously identified performance metric is met. 

• Provide metrics, methods, and tools to VSST for integration. 

• Investigate diagnostic and/or prognostic algorithm with respect to: (1) verifiability; (2) ability to 
distill varying degrees of knowledge of underlying physics; (3) ability to process varying degrees 
of knowledge about uncertainty 
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SSAT Project Technical Challenges 
Annual Performance Goal (APG) 



EXAMPLE (FY11/FY12) 

Data Mining - Scalable anomaly detection on heterogeneous data 


• Description: Development of a scalable algorithm for anomaly 
detection on data consisting of discrete and continuous sequences 
as well as text reports that have been matched up (i.e., are from the 
same flight). 


• Metric/Exit Criteria: Algorithm that identifies at least three 
anomalies (in real flight data) validated by an expert to be statistical 
anomalies. Run time should be nominally no more than 50% 
greater than the run time for the fastest algorithm that runs on only 
discrete and continuous sequences. 
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Mining Heterogeneous Data is the Key 




Primary Source: Radar data 
Can answer what happened in 
the National Airspace during 
Aviation Safety Incident (in 
preparation) 


Primary Source: Humans 
Can answer why 
an Aviation Safety Incident 
happened 


Primary Source: Aircraft 
Can answer what happened in 
during an Aviation Safety 
Incident 


Sample Text 
Report 

JUST PRIOR TO 
TOUCHDOWN, 
LAX TWR TOLD 
US TO GO 
AROUND 
BECAUSE OF 
THE ACFTIN 
FRONT OF US. 



Knowledge Dissemination 




Journal Articles 44 


NASA Technical Manuscripts 4 

Book Chapters & Contractor Reports 16 


Books 2 

DASHIink Downloads (Papers, Code, Approximately 3000 

and Data) downloads per month 


8 Awards at Major International 
Conferences: 

• IEEE International Conference on 
Data Mining 

• IEEE International Conference on 
Systems, Man, and Cybernetics 

• Prognostics and 

Health Management Society 

• Surface Mount 
Technology Association 

• Autotestcon 


link ^ 




| Register 

PROJECTS 


RESOURCES 


MEMBERS 



A web-based collaboration tool for those 


interested in data mining and systems 
health 


LEARN MORE 


Research Areas 

Leam about our research 
fields, goals and their 
associated projects. 


Projects 

See what others in the 
community are working 
on. Join or start your own. 


Resources 

Available data sets, 
algorithms, and 
publications FREE to 
download 
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Impact of the SSAT Project 



SSAT and IVHM influenced 
the design of the Central 
Maintenance Computer of 
the 787, Embraer, and other 
major jets. 

Transfer of ADEPT Software 
to Gulfstream to help design 
and analyze new concepts for 
controlling system functions. 

Fatigue Risk Management 
Studies at EasyJet and 
Onera are underway with 
both NASA Technical 
Reports published. 

Southwest Airlines utilizing 
data mining to improve 
safety of current operations. 




Health 


anaaement 


Chapman & Hall/CRC 

Data Mining and Knowledge Discovery Series 


Machine Learning and 

o 


ivnowiecige uiscovery 
for Engineering Systems 


ishok/N. Srivastava 
^ ^ Tiawei Han 


142 Conference Papers 
48 Journal Papers 
10 NASA Technical Manuscripts 
16 Book Chapters + Reports 
4 Invention Disclosures 
2 Books 


• Top 10 Data Mining Case 
Study at IEEE 

ICDM Conference 

• 8 Best Paper Awards 
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THANK YOU 
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